Skip to content

Plan and wire 10TB scale operating layer - #137

Merged
SonAIengine merged 5 commits into
mainfrom
scale-operating-layer-plan
Jul 3, 2026
Merged

Plan and wire 10TB scale operating layer#137
SonAIengine merged 5 commits into
mainfrom
scale-operating-layer-plan

Conversation

@SonAIengine

Copy link
Copy Markdown
Contributor

Summary

  • Add the v0.31 10TB scale operating-layer plan and current DeepSeek 50-query baseline notes.
  • Add scale-out indexing contracts plus local StorageCandidateProvider and InProcessIndexRouter implementations.
  • Wire optional index_router support into EvidenceSearch and SynapticGraph while keeping the default StorageBackend path unchanged.
  • Add tests for provider filtering/query variants, router dedupe diagnostics, EvidenceSearch routed seeds, and SynapticGraph router cache wiring.

Validation

  • uv run ruff check src/synaptic/indexing.py src/synaptic/extensions/evidence_search.py src/synaptic/graph.py src/synaptic/__init__.py src/synaptic/__init__.pyi tests/test_indexing_contracts.py tests/test_evidence_search.py tests/test_graph.py
  • uv run pytest tests/test_indexing_contracts.py tests/test_evidence_search.py tests/test_graph.py tests/test_agent_tools.py tests/test_agent_search.py tests/test_memory_operating_layer.py tests/test_router.py → 237 passed
  • uv run pytest → 1629 passed, 67 skipped
  • import smoke for StorageCandidateProvider / InProcessIndexRouter
  • DeepSeek 50-query current baseline artifact: 31/50 reach, mean first relevant turn 1.16, empty calls 6

@SonAIengine

Copy link
Copy Markdown
Contributor Author

Updated the scale operating layer PR with two concrete provider implementations:

  • Added OpenSearchCandidateProvider for lexical candidate ids with workspace/ACL/document/source/time/property filters.
  • Added QdrantCandidateProvider for vector candidate ids with payload filters and scored metadata.
  • Kept both providers dependency-injected, so core imports do not require OpenSearch or Qdrant runtime packages.
  • Updated PLAN-v0.31 near-term TODO to move the next focus to durable ingestion job/index health persistence and 100GB/1TB rehearsal.

Validation:

  • uv run ruff check src/synaptic/indexing.py src/synaptic/init.py src/synaptic/init.pyi tests/test_indexing_contracts.py
  • uv run pytest tests/test_indexing_contracts.py
  • uv run pytest tests/test_evidence_search.py tests/test_graph.py tests/test_router.py tests/test_agent_tools.py tests/test_agent_search.py
  • uv run pytest -> 1634 passed, 67 skipped
  • GitHub CI lint/test passing on c6d9274

@SonAIengine

Copy link
Copy Markdown
Contributor Author

Added the durable operational-state foundation as well:

  • MemoryBackend and SQLiteBackend now implement IngestionJobStore for stage/status/attempt/error/version/properties roundtrip.
  • MemoryBackend and SQLiteBackend now implement IndexHealthBackend for provider lag reports, pending counts, failed jobs, p95 latency, and provider properties.
  • PLAN-v0.31 now marks provider persistence as implemented and moves the next TODO to idempotent ingestion orchestration plus tombstone/replay flow.

Validation for this commit:

  • uv run ruff check src/synaptic/backends/memory.py src/synaptic/backends/sqlite.py tests/test_indexing_contracts.py
  • uv run pytest tests/test_indexing_contracts.py tests/test_backend_memory.py tests/test_backend_sqlite.py -> 82 passed
  • uv run pytest tests/test_memory_operating_layer.py tests/test_memory_operating_poc.py tests/test_document_ingester.py tests/test_evidence_search.py tests/test_graph.py -> 161 passed
  • GitHub CI lint/test passing on 2b1db24

@SonAIengine
SonAIengine merged commit a959b90 into main Jul 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant